Information processing device that projects drawing information according to ratio of distances, information processing method, and recording medium

ABSTRACT

An information processing device comprising a projection control unit that controls projection of drawing information, the projection being performed by a projection device, on a basis of ratio information between a first distance and a second distance, the first distance being acquired from sensing information regarding a drawing tool and being from a reference point to the drawing tool, and the second distance being from the reference point to an intersection point between a straight line connecting the reference point with the drawing tool and a calibration plane.

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2020/000598 filed on Jan. 10, 2020, which claimspriority benefit of Japanese Patent Application No. JP 2019-024870 filedin the Japan Patent Office on Feb. 14, 2019. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a recording medium.

BACKGROUND ART

In recent years, there has emerged a drawing system in which, when theuser moves a drawing tool on a drawing surface in a real space, aprojection device such as a projector projects the information such as aline indicating the trail of the drawing tool (hereinafter also referredto as drawing information) on the drawing surface. For example, PatentDocument 1 listed below discloses a technology for improving userconvenience in such a drawing system.

Such a drawing system typically includes an imaging device and aprojection device, and performs a calibration in advance for generatinga projective transformation matrix H between the imaging device and theprojection device using a predetermined plane for calibration(hereinafter also referred to as a calibration plane). Then, by usingsuch a projective transformation matrix H to project the drawinginformation, the drawing system is allowed to ensure that the positionof the drawing tool on the calibration plane matches the position of thedrawing information. As a result, the user can comfortably perform adrawing operation as if the user actually writes to a real space with apen or the like.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2014-134613

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Typically, a calibration is performed by using the drawing surface as acalibration plane. As a result, the position of the drawing tool on thedrawing surface and the position of drawing information projected on thedrawing surface are matched. However, when the drawing surface and thecalibration plane differ from each other, the position of the drawingtool on the drawing surface and the position of drawing informationprojected on the drawing surface may be mismatched. This is because acalibration is performed merely for the purpose of suppressing amismatch of positions on the calibration plane.

Thus, the present disclosure provides a mechanism capable of suppressingthe occurrence of a mismatch between the position of a drawing tool on adrawing surface and the position of drawing information projected on thedrawing surface in a case where the calibration plane and the drawingsurface differ from each other.

Solutions to Problems

According to the present disclosure, there is provided an informationprocessing device including: a projection control unit that controlsprojection of drawing information, the projection being performed by aprojection device, on the basis of ratio information between a firstdistance and a second distance, the first distance being acquired fromsensing information regarding a drawing tool and being from a referencepoint to the drawing tool, and the second distance being from thereference point to an intersection point between a straight lineconnecting the reference point with the drawing tool and a calibrationplane.

Furthermore, according to the present disclosure, there is provided aninformation processing method including: controlling projection ofdrawing information, the projection being performed by a projectiondevice, on the basis of ratio information between a first distance and asecond distance, the first distance being acquired from sensinginformation regarding a drawing tool and being from a reference point tothe drawing tool, and the second distance being from the reference pointto an intersection point between a straight line connecting thereference point with the drawing tool and a calibration plane.

Furthermore, according to the present disclosure, there is provided arecording medium recording a program that causes a computer to functionas: a projection control unit that controls projection of drawinginformation, the projection being performed by a projection device, onthe basis of ratio information between a first distance and a seconddistance, the first distance being acquired from sensing informationregarding a drawing tool and being from a reference point to the drawingtool, and the second distance being from the reference point to anintersection point between a straight line connecting the referencepoint with the drawing tool and a calibration plane.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for providing an overview of a drawing systemaccording to an embodiment of the present disclosure.

FIG. 2 is a diagram illustrating an example of a captured image capturedby an imaging unit in the state illustrated in FIG. 1 .

FIG. 3 is a diagram schematically showing a positional relationshipamong elements involved in the example illustrated in FIG. 1 .

FIG. 4 is a block diagram illustrating an example of a functionalconfiguration of the drawing system according to the present embodiment.

FIG. 5 is a diagram for explaining an example of a calibration with anauxiliary board according to the present embodiment.

FIG. 6 is a diagram for explaining an example of a map according to thepresent embodiment.

FIG. 7 is a flowchart showing an example flow of a preliminary processperformed by the drawing system according to the present embodiment.

FIG. 8 is a flowchart showing an example flow of a projection controlprocess performed by the drawing system according to the presentembodiment.

FIG. 9 is a diagram for explaining an example use of the drawing systemaccording to the present embodiment.

FIG. 10 is a diagram illustrating an example of a captured imagecaptured by the imaging unit in the state illustrated in FIG. 9 .

FIG. 11 is a diagram for explaining an example of projection control bya drawing system according to a first modification.

FIG. 12 is a block diagram illustrating an example of a functionalconfiguration of a drawing system according to a second modification.

FIG. 13 is a block diagram illustrating an example of a functionalconfiguration of a drawing system according to a third modification.

FIG. 14 is a diagram schematically showing a positional relationshipamong elements involved in the third modification.

FIG. 15 is a diagram for providing an overview of a drawing systemaccording to a fourth modification.

FIG. 16 is a diagram illustrating a positional relationship amongangular fields of view and center points of angular fields of view on afirst calibration plane illustrated in the example in FIG. 15 .

FIG. 17 is a diagram illustrating a positional relationship amongangular fields of view and center points of angular fields of view on asecond calibration plane illustrated in the example in FIG. 15 .

FIG. 18 is a diagram illustrating a relative positional relationshipbetween center points of angular fields of view on each of the firstcalibration plane and the second calibration plane illustrated in theexample in FIG. 15 .

FIG. 19 is a diagram for explaining an example use of the drawing systemaccording to the present modification.

FIG. 20 is a block diagram illustrating an example hardwareconfiguration of an information processing device according to thepresent embodiment.

MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the present disclosure will now be described indetail with reference to the accompanying drawings. Note that, in thedescription and the drawings, components having substantially the samefunctions and configurations are denoted by the same reference numeralsand duplicate descriptions are omitted.

Note that descriptions will be provided in the order shown below.

1. Introduction

2. Example configuration

3. Technical features

3.1. Acquiring position of digital pen

3.2. Preliminary process

3.3. Projection control

4. Modifications

4.1. First modification

4.2. Second modification

4.3. Third modification

4.4. Fourth modification

5. Example hardware configuration

6. Conclusion

1. INTRODUCTION

FIG. 1 is a diagram for providing an overview of a drawing system 1according to an embodiment of the present disclosure. As illustrated inFIG. 1 , the drawing system 1 includes an imaging unit 10, a projectionunit 20, and a digital pen 50.

The imaging unit 10 is an imaging device that captures an image of areal space. The imaging unit 10 includes a lens system, a drive system,and an imaging element for an RGB camera or the like to capture images(still images or moving images). The imaging unit 10 captures an imageof a real space included in an imaging angular field of view 11. Theimaging angular field of view 11, which refers to an imaging range, isdefined by the position of the imaging unit 10 (hereinafter alsoreferred to as a camera position C), the imaging direction, and theangle of an imaging range with the imaging direction being a centralaxis. An image captured by the imaging unit 10 is also referred to as acaptured image.

The projection unit 20 is a projection device that projects an image onsome place in the real space. The projection unit 20 projects an imagein a real space included in a projection angular field of view 21. Theprojection angular field of view 21, which refers to a projectablerange, is defined by the position of the projection unit 20 (hereinafteralso referred to as a projector position P), the projecting direction,and the angle of a projectable range with the projecting direction beinga central axis. An image projected by the projection unit 20 is alsoreferred to as a projected image. Note that it is assumed that aprojected image is an image projected on the entire projection angularfield of view 21, and the drawing information corresponding to useroperation of the digital pen 50 is mapped and projected in the projectedimage.

The digital pen 50 is a drawing tool in which a light-emitting part suchas an infrared (IR) part, a light-emitting diode (LED), or the like ismounted on the pen tip. The light-emitting part emits light when, forexample, a button, a switch, or the like disposed on the digital pen 50is operated, the pen tip is pressed against a contact surface, or thepen is shaken. In addition, the digital pen 50 may transmit apredetermined command based on a user operation of a button or a switchdisposed on the digital pen 50, movement of the pen, or the like to aprojection control unit 40, which is described later.

In a case where the imaging angular field of view 11 and the projectionangular field of view 21 are different as shown in FIG. 1 , acalibration for generating a projective transformation matrix H betweenthe imaging unit 10 and the projection unit 20 is performed in advanceby using a calibration plane S. In the example illustrated in FIG. 1 , atabletop surface 60 is used as the calibration plane S. Then, when theuser moves the digital pen 50 on a drawing surface S′, an image thereofis captured by the imaging unit 10, and the drawing informationrepresenting the trail of the digital pen 50 as a line is projected ontothe drawing surface S′ by the projection unit 20. For example, thedrawing information is projected onto the position of the digital pen 50(more technically, the position of the bright spot of the pen tip). Inthe example illustrated in FIG. 1 , a floor surface 62 included in theprojection angular field of view 21 except the tabletop surface 60 isused as the drawing surface S′.

Note that the calibration plane S is a plane including the plane on asurface of the calibration target object and a plane obtained byvirtually extending the plane on the surface of the calibration targetobject. In the example illustrated in FIG. 1 , the calibration targetobject is the tabletop surface 60, and the calibration plane S includesthe tabletop surface 60 and a plane obtained by virtually extending thetabletop surface 60. It is assumed here that the calibration plane S andthe drawing surface S′ are parallel.

In the example illustrated in FIG. 1 , it is assumed that the digitalpen 50 is located at a position d′ on the drawing surface S′. Inaddition, the intersection point between the straight line that connectsthe camera position C with the position d′ of the digital pen 50 and thecalibration plane S is denoted as a position d. In this case, thecoordinates of the digital pen 50 in a captured image are the sameregardless of whether the digital pen 50 is located at the position d onthe calibration plane S or the digital pen 50 is located at the positiond′ on the drawing surface S′. This matter is explained below withreference to FIG. 2 .

FIG. 2 is a diagram illustrating an example of a captured image capturedby the imaging unit 10 in the state illustrated in FIG. 1 . The capturedimage shows the state of the real space included in the imaging angularfield of view 11 as seen from the camera position C. The captured imageshows the floor surface 62 and the tabletop surface 60 in the imagingangular field of view 11. In the captured image, the tabletop surface 60is the calibration plane S, while the floor surface 62 included in theprojection angular field of view 21 except the tabletop surface 60 isthe drawing surface S′. Since the captured image is a two-dimensionalimage, the distance along a depth direction of the digital pen 50 asseen from the imaging unit 10 is unknown. Therefore, the coordinates ofthe digital pen 50 in the captured image are the same regardless ofwhether the digital pen 50 is located at the position d on thecalibration plane S or the digital pen 50 is located at the position d′on the drawing surface S′. That is, letting (x_(d), y_(d)) be thecoordinates of the position d in the captured image and letting (x_(d′),y_(d′)) be the coordinates of the position d′ in the captured image,(x_(d), y_(d))=(x_(d′), y_(d′)) is satisfied.

In a case where a projective transformation matrix H on the calibrationplane S is used as it is despite the fact that the distance along adepth direction from the imaging unit 10 to the calibration plane S isdifferent from the distance from the imaging unit 10 to the drawingsurface S′, a mismatch occurs between the position of the digital pen 50on the drawing surface S′ and the position of the drawing informationprojected on the drawing surface S′. This matter is explained below withreference to FIG. 3 .

FIG. 3 is a diagram schematically showing a positional relationshipamong elements involved in the example illustrated in FIG. 1 . FIG. 3illustrates the camera position C and the projector position P at thesame position on the assumption that the distance between the cameraposition C and the projector position P is negligibly small as comparedwith the distances therefrom to the calibration plane S and to thedrawing surface S′. As shown in FIG. 3 , the imaging angular field ofview 11 and the projection angular field of view 21 are different. Thewidth of the imaging angular field of view 11 on the calibration plane Sis represented by D₁, while the width of the imaging angular field ofview 11 on the drawing surface S′ is represented by D₂. Furthermore, thewidth of the projection angular field of view 21 on the calibrationplane S is represented by D₃, while the width of the projection angularfield of view 21 on the drawing surface S′ is represented by D₄.

According to laws of perspective, the size of a real object in acaptured image is scaled up or down in accordance with the distancealong a depth direction from the camera position C to the real object.For example, the ratio r_(C) of the scaling amount of a real object in acaptured image between the case where the real object is located on thecalibration plane S and the case where the real object is located on thedrawing surface S′ is simply defined by the mathematical expression (1).r _(C) =D ₂ /D ₁  (1)

Likewise, according to laws of perspective, the size of a real object ina projected image is scaled up or down in accordance with the distancealong a depth direction from the projector position P to the realobject. For example, the ratio r_(C) of the scaling amount of a realobject in a projected image between the case where the real object islocated on the calibration plane S and the case where the real object islocated on the drawing surface S′ is simply defined by the mathematicalexpression (2).r _(P) =D ₄ /D ₃  (2)

Then, in a case where the imaging angular field of view 11 and theprojection angular field of view 21 are different, a difference arisesbetween the scaling ratio r_(c) in a captured image and the scalingratio r_(P) in a projected image. The difference in scaling ratio is thecause of a mismatch occurring between the position of the digital pen 50on the drawing surface S′ and the position of the drawing informationprojected on the drawing surface S′.

As shown in FIG. 3 , given that the distance from the camera position Cto the position d is 1, the distance from the camera position C to theposition d′ is represented by a, where a>1. Suppose that the digital pen50 has been moved from the position d′ by a distance n on the drawingsurface S′. Then, the movement distance of the digital pen 50 in acaptured image is calculated in accordance with the mathematicalexpression (3), where n_(c) is the movement distance from the position don the calibration plane S. Then, on the basis of the movement distancen_(c), the drawing position of the drawing information is controlled.n _(c) =r _(c)×(1/a)×n  (3)

On the other hand, the movement distance of the digital pen 50 in aprojected image is calculated in accordance with the mathematicalexpression (4), where n_(p) is the movement distance from the position don the calibration plane S.n _(p) =r _(p)×(1/a)×n  (4)

Here, as described above, r_(c)≠r_(p) is established when the imagingangular field of view 11 and the projection angular field of view 21 aredifferent, and accordingly n_(c)≠n_(p) is established. Therefore, amismatch occurs between the position of the digital pen 50 on thedrawing surface S′ and the position of the drawing information projectedon the drawing surface S′.

Thus, the present disclosure proposes a technology for carrying out aprocess in accordance with the difference between the distance along adepth direction from the camera position C to the calibration plane Sand the distance therefrom to the drawing surface S′. More specifically,according to the proposed technology, a projective transformation matrixH′ specialized for the position d′ on the drawing surface S′ iscalculated in real time in accordance with the difference in distancealong a depth direction. Therefore, with respect to the position d′, thedifference in distance along a depth direction between the distance fromthe camera position C to the calibration plane S and the distancetherefrom to the drawing surface S′ is apparently eliminated, therebypreventing the occurrence of a difference between the above-describedscaling ratios r_(C) and r_(P). As a result, suppressing the occurrenceof a mismatch between the position of the digital pen 50 on the drawingsurface S′ and the position of the drawing information projected on thedrawing surface S′ can be achieved.

2. EXAMPLE CONFIGURATION

FIG. 4 is a block diagram illustrating an example of a functionalconfiguration of the drawing system 1 according to the presentembodiment. As illustrated in FIG. 4 , the drawing system 1 includes astorage unit 30 and a projection control unit 40, in addition to theimaging unit 10, the projection unit 20, and the digital pen 50 thathave been described with reference to FIG. 1 . Note that illustration ofthe digital pen 50 is omitted in FIG. 4 .

(1) Imaging Unit 10

The configuration of the imaging unit 10 is as described above. Theimaging unit 10 outputs a captured image to a position acquisition unit41.

(2) Projection Unit 20

The configuration of the projection unit 20 is as described above. Theprojection unit 20 projects a projected image under the control of anoutput control unit 44.

(3) Storage Unit 30

The storage unit 30 temporarily or permanently stores various types ofinformation for operations of the drawing system 1. For example, thestorage unit 30 stores a map M, which is described later.

(4) Projection Control Unit 40

The projection control unit 40, which functions as an arithmeticprocessing device and a control device, controls the overall operationin the drawing system 1 in accordance with various programs. Asillustrated in FIG. 4 , the projection control unit 40 includes theposition acquisition unit 41, a calibration unit 42, a map generationunit 43, and the output control unit 44. The position acquisition unit41, which has a function of acquiring the position of the digital pen50, outputs the information indicating the position of the digital pen50 to each of the calibration unit 42, the map generation unit 43, andthe output control unit 44. The calibration unit 42, which has afunction of performing a calibration, outputs a projectivetransformation matrix H generated through the calibration to the outputcontrol unit 44. The map generation unit 43, which has a function ofgenerating a map M as described later, outputs the generated map M tothe output control unit 44. The output control unit 44 has a function ofperforming projection control over a projected image on the basis of theposition of the digital pen 50, the projective transformation matrix H,and the map M. Note that the projection control unit 40 may include anyother function in addition to the functions fulfilled by thesecomponents.

3. TECHNICAL FEATURES

<3.1 Acquiring Position of Digital Pen>

The position acquisition unit 41 acquires the position of the digitalpen 50 on the basis of sensing information regarding the digital pen 50.The position of the digital pen 50 herein refers to a concept includingthe distance from the camera position C to the digital pen 50 and thecoordinates of the digital pen 50 in a captured image.

Acquiring Distance Along Depth Direction

Specifically, the position acquisition unit 41 acquires the distancefrom a reference point to the digital pen 50 on the basis of the sensinginformation. In the present embodiment, the sensing information is acaptured image taken by the imaging unit 10. In addition, the positionof the reference point is the position of the imaging unit 10 (that is,the camera position C). Note that the present embodiment regards thecamera position C and the projector position P as the same position onthe assumption that the distance between the camera position C and theprojector position P is negligibly small as compared with the distancestherefrom to the calibration plane S and to the drawing surface S′. Inother words, the position of the reference point may be regarded as theprojector position. Furthermore, the present embodiment assumes that theoptical axes of the imaging unit 10 and the projection unit 20 coincide.

The position acquisition unit 41 may acquire the distance from thecamera position C to the digital pen 50 on the basis of the size of aspecific portion of the digital pen 50, the portion appearing in acaptured image. Pursuant to laws of perspective, the size of a specificportion of the digital pen 50 appearing in a captured image is scaled upor down in accordance with the distance from the camera position C tothe digital pen 50. Therefore, the position acquisition unit 41 acquiresa smaller distance value as the size of a specific portion of thedigital pen 50 appearing in a captured image is larger, and acquires alarger distance value as the size is smaller.

The specific portion may be, for example, a light-emitting part. In thiscase, the position acquisition unit 41 acquires the distance from thecamera position C to the digital pen 50 on the basis of the size of theshape of light (for example, IR light or visible light) emitted from thelight-emitting part of the digital pen 50, the shape appearing in acaptured image. For example, in a case where the light-emitting part ofthe digital pen 50 is annular, the position acquisition unit 41 acquiresthe distance from the camera position C to the digital pen 50 on thebasis of the size (area, radius, or the like) of the circle of light(hereinafter also referred to as a light-emitting circle) emitted fromthe light-emitting part of the digital pen 50 appearing in a capturedimage. It is hereinafter assumed that the specific portion is alight-emitting circle and the distance from the camera position C to thedigital pen 50 is acquired on the basis of the radius of thelight-emitting circle.

The acquired distance may be an actual distance. For example, the actualdistance is acquired if the relationship between the radius of thelight-emitting circle of the digital pen 50 appearing in a capturedimage and the actual distance from the camera position C to the digitalpen 50 is defined in advance. Alternatively, the acquired distance maybe information representing a distance. For example, the radius of thelight-emitting circle of the digital pen 50 appearing in a capturedimage may be acquired as the information representing a distance.

Acquiring Coordinates in Captured Image

On the basis of the sensing information, the position acquisition unit41 acquires the coordinates of the digital pen 50 in a captured image.For example, on the basis of the coordinates of a bright spot of thedigital pen 50 appearing in a captured image, the position acquisitionunit 41 acquires the coordinates of the digital pen 50 in the capturedimage.

Supplementary Information

Note that a specific portion of the digital pen 50 is not limited to thelight-emitting circle. For example, in a case where a two-dimensionalcode such as a bar code is attached to the digital pen 50, the positionof the digital pen 50 may be acquired by using the two-dimensional code.

<3.2. Preliminary Process>

(1) Calibration

The calibration unit 42 performs a calibration. Specifically, withregard to a point on the calibration plane S, the calibration unit 42generates a projective transformation matrix H defining a relationshipbetween the coordinates of the point in a captured image captured by theimaging unit 10 and the coordinates of the point in a projected imageprojected by the projection unit 20.

For the calibration, the projection unit 20 projects a projected imageon the calibration plane S, and the imaging unit 10 captures theprojected image projected on the calibration plane S. Then, thecalibration unit 42 generates a projective transformation matrix H onthe basis of the coordinates of a plurality of known points in thecaptured image of the projected image projected on the calibration planeS. Such coordinates are acquired by the position acquisition unit 41 onthe basis of a captured image captured in a case where, for example, thedigital pen 50 is positioned at the plurality of known points. Theplurality of known points is, for example, vertices of a projectedimage.

In the example described with reference to FIG. 1 , since the projectionangular field of view 21 exceeds the tabletop surface 60, an image canbe drawn on the floor surface 62. In this case, when an attempt is madeto perform a calibration on the basis of vertices of a projected imageprojected on the whole projection angular field of view 21, it isdifficult to recognize vertices of the projected image projected on thecalibration plane S because such vertices exist in the air on thecalibration plane S. Then, an auxiliary board may be used to allowvertices of a projected image to be projected on the calibration planeS. This matter is explained below with reference to FIG. 5 .

FIG. 5 is a diagram for explaining an example of a calibration with theauxiliary board according to the present embodiment. The tabletopsurface 60 illustrated in FIG. 5 is a plane of a surface of thecalibration target object included in the calibration plane S. In thiscase, although an attempt is made to perform a calibration on the basisof vertices of a projected image projected on the whole projectionangular field of view 21, such vertices exist in the air on thecalibration plane S. Then, the user holds the auxiliary board 61 suchthat a surface of the auxiliary board 61 is located on a plane extendedfrom the tabletop surface 60. As a result, a vertex d₃ of a projectedimage is projected on the calibration plane S as illustrated in FIG. 5 ,making it possible to obtain a captured image in which the digital pen50 is positioned at the vertex d₃. The calibration unit 42 recognizesthe position of the vertex d₃ in the captured image by recognizing thecoordinates of the bright spot of the digital pen 50 in the capturedimage. A similar process is carried out on any other vertex, whereby thecalibration unit 42 can recognize the coordinates of each of thevertices d₁ to d₄ of a projected image appearing in a captured image.Note that the auxiliary board 61 is a flat tool that may include anymaterial and may have any shape.

(2) Generating Map M

The map generation unit 43 generates a map M that defines, with regardto each coordinate point in a captured image, the distance from thecamera position C to the calibration plane S. Since the calibrationplane S is a flat surface, it is assumed that the distance from thecamera position C to each coordinate point on the calibration plane Svaries one-dimensionally. Therefore, the map generation unit 43calculates, with respect to each coordinate point on a captured image,the distance from the camera position C to the calibration plane S onthe basis of the distances from the camera position C to a plurality ofpoints on the calibration plane S and, on the basis of the calculationresult, generates the map M.

The map M may be generated on the basis of the size of a specificportion of the digital pen 50 appearing in a captured image, in a casewhere the digital pen 50 is located at a plurality of points on thecalibration plane S. For example, the map M is generated on the basis ofthe size of a light-emitting circle in a case where the digital pen 50is located at a plurality of points on the calibration plane S. Theplurality of points used for generating the map M may be a plurality ofknown points used for the calibration as described above. In this case,the calibration and generation of the map M are completed by, forexample, simply locating the digital pen 50 at the vertices d₁ to d₄ ofa projected image appearing in a captured image, and thus the burden onthe user is reduced.

The distance defined in the map M may be an actual distance value orinformation representing the distance. Examples of the informationrepresenting the distance include the radius of the light-emittingcircle of the digital pen 50 appearing in a captured image. In thiscase, the map M defines the radius of the light-emitting circle of thedigital pen 50 expected to appear in a captured image, in a case wherethe digital pen 50 is located at a position on the calibration plane S,the position corresponding to each coordinate point in the capturedimage. A specific example of the generated map M is described below withreference to FIG. 6 .

FIG. 6 is a diagram for explaining an example of the map M according tothe present embodiment. As shown in FIG. 6 , r(d) represents the radiusof the light-emitting circle appearing in a captured image in a casewhere the digital pen 50 is located at a position d on the calibrationplane S. The map generation unit 43 calculates the radius r(d) of thelight-emitting circle at any coordinate point (x_(d), y_(d)) in acaptured image on the basis of the radii r(d₁) to r(d₄) of thelight-emitting circles appearing in the captured image in a case wherethe digital pen 50 is located at the vertices d₁ to d₄ of a projectedimage. As a result, the map M defining the radius r(d) of thelight-emitting circle at any coordinate point (x_(d), y_(d)) in acaptured image is generated.

(3) Process Flow

FIG. 7 is a flowchart showing an example flow of the preliminary processperformed by the drawing system 1 according to the present embodiment.As shown in FIG. 7 , first, the projection unit 20 projects a projectedimage on the entire projection angular field of view 21 (step S102).Then, the imaging unit 10 captures an image of the state where thedigital pen 50 is located at the vertices d₁ to d₄ of the projectedimage projected on the calibration plane S (S104). In this step, when avertex of the projected image exists in the air as described above withreference to FIG. 5 , the auxiliary board 61 is used to project thevertex of the projected image on the calibration plane S. Next, theposition acquisition unit 41 acquires, as the coordinates of each of thevertices d₁ to d₄, the coordinates of each of the bright spots of thedigital pens 50 appearing in the captured image in a case where thedigital pen 50 is located at the vertices d₁ to d₄ (S106). Then, theposition acquisition unit 41 acquires, as the information representingthe distances from the camera position C to the vertices d₁ to d₄, theradii r(d₁) to r(d₄) of the light-emitting circles of the digital pen 50in a case where the digital pen 50 is located at the vertices d₁ to d₄(step S108). Next, the calibration unit 42 generates a projectivetransformation matrix H on the calibration plane S on the basis of thecoordinates of each of the vertices d₁ to d₄ in the captured image (stepS106). Then, the map generation unit 43 generates the map M defining theradius r(d) of the light-emitting circle at any coordinate point (x_(d),y_(d)) in a captured image, on the basis of the coordinates of each ofthe vertices d₁ to d₄ and the radii r(d₁) to r(d₄) of the light-emittingcircles in the captured image (step S108).

<3.3. Projection Control>

The output control unit 44 controls projection of drawing information onthe basis of the position of the digital pen 50 (that is, thecoordinates in a captured image and the distance from the cameraposition C), the projective transformation matrix H, and the map M.Specifically, the output control unit 44 controls projection of thedrawing information performed by the projection unit 20 on the basis ofthe coordinates of the digital pen 50 on the drawing surface S′ (forexample, the coordinates of the digital pen 50 in a captured image).Moreover, the output control unit 44 controls the projection of thedrawing information performed by the projection unit 20 on the basis ofthe ratio information between a first distance and a second distance,the first distance being from the reference point to the digital pen 50,and the second distance being from the reference point to theintersection point between the straight line connecting the referencepoint with the digital pen 50 and the calibration plane S. Thecoordinates and the ratio information regarding the digital pen 50 onthe drawing surface S′ can be acquired from the sensing informationregarding the digital pen 50. The sensing information regarding thedigital pen 50 herein refers to the sensing information regarding thedigital pen 50 that is in drawing operation on the drawing surface S′.

In the examples illustrated in FIGS. 1 to 3 , the first distance is thedistance from the camera position C to the position d′ of the digitalpen 50 on the drawing surface S′. The first distance can also beregarded as the distance from the camera position C to the position ofthe digital pen 50 in drawing operation. The first distance is acquiredby the position acquisition unit 41. The first distance is acquired onthe basis of, for example, the radius r(d′) of the light-emitting circleof the digital pen 50 appearing in a captured image.

In the examples illustrated in FIGS. 1 to 3 , the second distance is thedistance from the camera position C to the position d of theintersection point between the straight line connecting the cameraposition C with the position d′ of the digital pen 50 and thecalibration plane S. The second distance is acquired with reference tothe map M. Specifically, the output control unit 44 acquires, as thesecond distance, the distance defined in the map M with respect to thecoordinates (x_(d), y_(d)) corresponding to the coordinates (x_(d′),y_(d′)) of the digital pen 50 included in a captured image. In otherwords, the output control unit 44 acquires, as the second distance, thedistance indicated by the radius r(d) of the light-emitting circle thatis expected to appear in a captured image if the digital pen 50 islocated at the position d on the calibration plane S. Note that (x_(d),y_(d)=(x_(d′), y_(d′)) is satisfied.

The ratio information between the first distance and the second distancemay be acquired on the basis of the ratio between the first distance andthe second distance. For example, the output control unit 44 controlsthe projection of drawing information performed by the projection unit20 on the basis of the ratio between the first distance and the seconddistance. In the examples illustrated in FIGS. 1 to 3 , for example,assuming that the distance from the camera position C to the position dillustrated in FIG. 3 is 1, the ratio between the first distance and thesecond distance is given as a distance a from the camera position C tothe position d′. The ratio a may be regarded as the ratio between r(d)and r(d′) and calculated in accordance with the mathematical expression(5):a=r(d)/r(d′)  (5)

where a>1. Note that the ratio information between the first distanceand the second distance may be acquired on the basis of, for example,the difference between the first distance and the second distance.

The output control unit 44 controls the coordinates of the drawinginformation in a projected image projected by the projection unit 20.Specifically, first, the output control unit 44 calculates a projectivetransformation matrix H′ at the position d′ on the drawing surface S′ onthe basis of the ratio a between the first distance and the seconddistance. Then, the output control unit 44 controls the coordinates ofthe drawing information in the projected image by using the correctedprojective transformation matrix H′ and the coordinates (x_(d′), y_(d′))of the digital pen 50 in a captured image. That is, the output controlunit 44 converts the coordinates of the position d′ in the capturedimage into the coordinates in the projected image by using the correctedprojective transformation matrix H′, places the drawing information atthe converted coordinates in the projected image, and causes theprojection unit 20 to project the drawing information. Note that thecorrected projective transformation matrix H′ is calculated inaccordance with the mathematical expression (6).H′=aH  (6)

As described above, the projective transformation matrix H obtainedduring calibration is not used as it is, but the corrected projectivetransformation matrix H′, which has been corrected in accordance withthe distance difference between the first distance and the seconddistance with respect to the position d′, is used. As a result,suppressing the occurrence of a mismatch between the position d′ of thedigital pen 50 on the drawing surface S′ and the position of the drawinginformation projected on the drawing surface S′ can be achieved.

Process Flow

With reference to FIG. 8 , the following describes an example of a flowof the projection control process described above.

FIG. 8 is a flowchart showing an example flow of the projection controlprocess performed by the drawing system 1 according to the presentembodiment. As shown in FIG. 8 , first, the position acquisition unit 41acquires the coordinates (x_(d′), y_(d′)) of the digital pen 50 and theradius r(d′) of the light-emitting circle in a captured image on thebasis of the captured image of the digital pen 50 on the drawing surfaceS′ (step S202). Then, the output control unit 44 refers to the map M toacquire the radius r(d) of the light-emitting circle at the coordinates(x_(d), y_(d))=(x_(d′), y_(d′)) (step S204). Next, the output controlunit 44 calculates the ratio a between the first distance and the seconddistance in accordance with the mathematical expression (5) above (stepS206). Then, the output control unit 44 calculates a projectivetransformation matrix H′ at the position d′ in accordance with themathematical expression (6) above (step S208). Then, the output controlunit 44 calculates the coordinates of the drawing information in aprojected image by using the corrected projective transformation matrixH′, and causes the projection unit 20 to project the drawing information(step S210).

Note that the projection control process shown in FIG. 8 may berepeatedly performed during a time period after the bright spot of thedigital pen 50 appears in the imaging angular field of view 11 until thebright spot disappears therefrom.

Supplementary Information

Note that the foregoing has described projection control performed in acase where the drawing surface S′ is farther from the camera position Cthan the calibration plane S. The following describes projection controlperformed in a case where the drawing surface S′ is closer to the cameraposition C than the calibration plane S with reference to FIGS. 9 and 10.

FIG. 9 is a diagram for explaining an example use of the drawing system1 according to the present embodiment. FIG. 10 is a diagram illustratingan example of a captured image captured by the imaging unit 10 in thestate illustrated in FIG. 9 . As illustrated in FIG. 9 , a doll 64 isplaced on the tabletop surface 60, and a surface of the doll 64 servesas the drawing surface S′. In this case, a captured image shown in FIG.10 is obtained.

The projection control performed in a case where the drawing surface S′is closer to the camera position than the calibration plane S is similarto the projection control performed in a case where the drawing surfaceS′ is farther from the camera position than the calibration plane S.Note that, however, the ratio a calculated in accordance with themathematical expression (5) above satisfies 0<a<1.

Furthermore, as illustrated in FIG. 9 , the surface of the doll 64 iscurved. The drawing surface S′ may be a surface of a three-dimensionalobject in any shape having a curved surface and the like. However, theabove-described algorithm assumes that the calibration plane S and thedrawing surface S′ are parallel. For this reason, in a case where thedrawing surface S′ is curved, the area of the position d′ of the digitalpen 50 in the drawing surface S′ is segmentalized so that the area isregarded as parallel to the calibration plane S, and then theabove-described algorithm is applied.

4. MODIFICATIONS

<4.1. First Modification>

A first modification is an example in which the output control unit 44controls not only the coordinates as described above but also otheritems of the drawing information in a projected image.

Controlling Size of Drawing Information

The output control unit 44 may control the size of the drawinginformation in a projected image projected by the projection unit 20.Specifically, the output control unit 44 controls the size of thedrawing information on the basis of the ratio a between the firstdistance and the second distance. The size of the drawing informationis, for example, the thickness of a line drawn in accordance with thetrail of the digital pen 50. For example, the output control unit 44performs control to make the line thickness uniform. This matter isexplained below with reference to FIG. 11 .

FIG. 11 is a diagram for explaining an example of projection control bythe drawing system 1 according to the first modification. As illustratedin FIG. 11 , in a case where the digital pen 50 is located at theposition d on the calibration plane S, the apparent size (for example,in units of meters) of the drawing information projected on thecalibration plane S is represented by W. In addition, the size of thedrawing information in terms of data (for example, in units of pixels)having the size W projected in a projected image on the calibrationplane S is represented by b_(d). Furthermore, as illustrated in FIG. 11, in a case where the digital pen 50 is located at the position d′ onthe drawing surface S′, the apparent size of the drawing informationprojected on the drawing surface S′ is represented by W′. In addition,the size of the drawing information in terms of data having the size W′projected in a projected image on the drawing surface S′ is representedby b_(d′). If b_(d′)=b_(d) is satisfied, W′=aW is established. Then, theoutput control unit 44 controls the size b_(d′) of the drawinginformation in terms of data in accordance with the mathematicalexpression (7).b _(d′) =b _(d) /a  (7)

As a result, it is made possible to make the apparent thickness of aline drawn in accordance with the trail of the digital pen 50 uniformregardless of the distance between the camera position C and the digitalpen 50.

Controlling Brightness of Drawing Information

The output control unit 44 may control the brightness of the drawinginformation in a projected image projected by the projection unit 20.Specifically, the output control unit 44 controls the brightness of thedrawing information on the basis of the ratio a between the firstdistance and the second distance. The brightness of the projecteddrawing information in terms of data in a case where the digital pen 50is located at the position d on the calibration plane S is representedby v, and the brightness of the projected drawing information in termsof data in a case where the digital pen 50 is located at the position d′on the drawing surface S′ is represented by v′. As a precondition, theapparent brightness of the projected drawing information is reduced inproportion to the distance from the projection unit 20. Then, the outputcontrol unit 44 controls the brightness v_(d′) of the drawinginformation in terms of data in accordance with the mathematicalexpression (8) using a coefficient e that is calculated on the basis ofthe ratio a.v _(d′) =e×v _(d)  (8)

As a result, it is made possible to make the apparent brightness of aline drawn in accordance with the trail of the digital pen 50 uniformregardless of the distance between the camera position C and the digitalpen 50.

Controlling Color of Drawing Information

The output control unit 44 may control the color of the drawinginformation in a projected image projected by the projection unit 20.Specifically, the output control unit 44 controls the color of thedrawing information on the basis of the ratio a between the firstdistance and the second distance. For example, the apparent colorsaturation of the projected drawing information is reduced in proportionto the distance from the projection unit 20. Thus, the output controlunit 44 uses a coefficient that is calculated on the basis of the ratioa to control the color saturation of the drawing information in terms ofdata, as with the brightness control in accordance with the mathematicalexpression (8) above.

As a result, it is made possible to make the apparent color saturationof a line drawn in accordance with the trail of the digital pen 50uniform regardless of the distance between the camera position C and thedigital pen 50.

Combination

The number of control items is not limited to one, and thus the outputcontrol unit 44 may simultaneously control two or more of thecoordinates, the size, the brightness, and the color of the drawinginformation in a projected image.

<4.2. Second Modification>

A second modification is an example in which the distance from thereference point to the digital pen 50 is acquired on the basis of aresult of distance measurement by a distance measuring sensor.

FIG. 12 is a block diagram illustrating an example of a functionalconfiguration of a drawing system 1 according to the secondmodification. As illustrated in FIG. 12 , the drawing system 1 accordingto the present modification further includes a distance measuring unit12, in addition to the functional configuration illustrated in FIG. 3 .The following describes the distance measuring unit 12 and functions ofthe other components characteristic of the present modification.

The distance measuring unit 12 is a sensor device that senses thedistance to the target object. For example, the distance measuring unit12 senses the distance to the digital pen 50 and outputs the sensinginformation (that is, the distance information) obtained as a result ofthe sensing to the position acquisition unit 41. The distance measuringunit 12 may be implemented by, for example, an ultrasonic distancemeasuring sensor, a time-of-flight (ToF) type image sensor, a stereocamera, or the like.

The position acquisition unit 41 acquires the distance from the cameraposition C to the digital pen 50 on the basis of the sensing informationprovided by the distance measuring unit 12. A calibration between theimaging unit 10 and the distance measuring unit 12 is preferablyperformed in advance. The map M is generated on the basis of thedistance to the digital pen 50 as provided by the distance measuringunit 12 in a case where the digital pen 50 is located at a plurality ofpoints on the calibration plane S. The plurality of points used forgenerating the map M may be a plurality of known points used for thecalibration. In this case, a calibration is performed on the basis ofthe coordinates of the digital pen 50 in a captured image in a casewhere the digital pen 50 is located at vertices d₁ to d₄ of a projectedimage, and the map M is generated on the basis of the distanceinformation obtained by the distance measuring unit 12.

Note here that, when the map M is generated on the basis of the distanceinformation obtained by the distance measuring unit 12, the distancedefined in the map M is an actual distance. The map M defines the actualdistance from the camera position C to the digital pen 50 in a casewhere the digital pen 50 is located at a position on the calibrationplane S corresponding to each coordinate point in a captured image.

The projection control by the output control unit 44 according to thepresent modification is basically similar to the projection control onthe basis of the radius of a light-emitting circle described above.However, the first distance is sensed as an actual value by the distancemeasuring unit 12.

Note that the projective transformation matrix H and the map M may bepreset as long as the positional relationship among the imaging unit 10,the projection unit 20, the distance measuring unit 12, and thecalibration plane S is fixed. In this case, calibration and generationof the map M are omitted.

<4.3. Third Modification>

A third modification is an example in which the projection control isperformed by using a three-dimensional position of the digital pen 50instead of a captured image of the digital pen 50.

(1) Example Configuration

FIG. 13 is a block diagram illustrating an example of a functionalconfiguration of a drawing system 1 according to the third modification.As illustrated in FIG. 13 , the drawing system 1 according to thepresent modification includes an inertial sensor unit 16, the projectionunit 20, the storage unit 30, and the projection control unit 40. Inaddition, the projection control unit 40 includes the positionacquisition unit 41, the calibration unit 42, and the output controlunit 44.

The inertial sensor unit 16 senses inertial information regarding thedigital pen 50. The inertial sensor unit 16, which includes anacceleration sensor and a gyro sensor, is mounted on the digital pen 50.The inertial sensor unit 16 senses the acceleration and the angularvelocity of the digital pen 50, and transmits the sensed accelerationand angular velocity to the position acquisition unit 41.

The following describes technical features of the present modificationwith reference to FIG. 14 . FIG. 14 is a diagram schematically showing apositional relationship among elements involved in the thirdmodification.

(2) Acquiring Position of Digital Pen 50

The position acquisition unit 41 acquires the position of the digitalpen 50 on the basis of sensing information regarding the digital pen 50.The sensing information here refers to the acceleration and the angularvelocity obtained by the inertial sensor unit. The position acquisitionunit 41 acquires the three-dimensional position of the digital pen 50through calculation by an inertial navigation system (IMS). Note that,however, the three-dimensional position acquired here is a relativethree-dimensional position with respect to any certain reset origin Oshown in FIG. 13 .

(3) Calibration

In the present modification, the calibration is performed by using twocalibration planes: a first calibration plane S1 and a secondcalibration plane S₂. For example, the first calibration plane S₁ is thetabletop surface 60 illustrated in FIG. 1 , the second calibration planeS₂ is the floor surface 62 illustrated in FIG. 1 , and the drawingsurface S′ is a surface of any other real object. The reference point inthe present modification is the projector position P.

The calibration unit 42 acquires the three-dimensional position of thedigital pen 50 in a case where the digital pen 50 is located at aplurality of known points on each of projected images projected on thefirst calibration plane S₁ and the second calibration plane S₂. Forexample, the calibration unit 42 acquires the three-dimensionalpositions of the plurality of known points in a projected imageprojected on the first calibration plane S₁. Such three-dimensionalpositions are acquired by the position acquisition unit 41 on the basisof, for example, the inertial information provided in a case where thedigital pen 50 is located at the plurality of known points. Theplurality of known points is, for example, vertices d₁₋₁ to d₁₋₄ of aprojected image. Similarly, the calibration unit 42 acquires thethree-dimensional positions of the plurality of known points in aprojected image projected on the second calibration plane S₂. Suchthree-dimensional positions are acquired by the position acquisitionunit 41 on the basis of, for example, the inertial information providedin a case where the digital pen 50 is located at the plurality of knownpoints. The plurality of known points is, for example, vertices d₂₋₁ tod₂₋₄ of a projected image.

Then, the calibration unit 42 generates a projective transformationmatrix for converting the acquired three-dimensional position on thefirst calibration plane S₁ or the second calibration plane S₂ into thecoordinates in a projected image projected by the projection unit 20.Specifically, first, the calibration unit 42 geometrically calculatesthe projector position P on the basis of the acquired three-dimensionalpositions of the vertices d₁₋₁ to d₁₋₄ and the acquiredthree-dimensional positions of the vertices d₂₋₁ to d₂₋₄. Then, thecalibration unit 42 calculates a projective transformation matrix H₁with regard to the first calibration plane S₁ on the basis of theprojector position P and the three-dimensional positions of the verticesd₁₋₁ to d₁₋₄. Alternatively, the calibration unit 42 calculates aprojective transformation matrix H₂ with regard to the secondcalibration plane S₂ on the basis of the projector position P and thethree-dimensional positions of the vertices d₂₋₁ to d₂₋₄.

(4) Projection Control

The output control unit 44 controls projection of the drawinginformation by the projection unit 20, on the basis of thethree-dimensional position of the digital pen 50 in a case where thedigital pen 50 is located at a plurality of known points on each ofprojected images projected on the first calibration plane S₁ and thesecond calibration plane S₂ and on the basis of the three-dimensionalposition of the digital pen 50 in drawing operation. Specifically, theoutput control unit 44 controls projection of the drawing informationperformed by the projection unit 20, on the basis of the projectivetransformation matrix H obtained through calibration and thethree-dimensional position of the digital pen 50 in drawing operation.

Let d′ be the three-dimensional position of the digital pen 50 indrawing operation. In addition, let position d₁ be the position of theintersection point between the straight line that connects the projectorposition P with the position d′ of the digital pen 50 and the firstcalibration plane S₁. Furthermore, let position d₂ be the position ofthe intersection point between the straight line that connects theprojector position P with the position d′ of the digital pen 50 and thesecond calibration plane S₂.

First, the output control unit 44 calculates a ratio a₁ or a ratio a₂between the distance from the projector position P to the position d′ ofthe digital pen 50 and the distance from the projector position P to theposition d₁ or the position d₂. For example, as illustrated in FIG. 14 ,assuming that the distance from the projector position P to the positiond₁ is 1, the ratio a₁ between the first distance and the second distancewith regard to the first calibration plane S₁ is given as a distance a₁from the projector position P to the position d′. On the other hand,although not illustrated in FIG. 14 , assuming that the distance fromthe projector position P to the position d₂ is 1, the ratio a₂ betweenthe first distance and the second distance with regard to the secondcalibration plane S₂ is given as a distance a₂ from the projectorposition P to the position d′.

Then, the output control unit 44 calculates a corrected projectivetransformation matrix H₁′ or H₂′ in accordance with the mathematicalexpression (6) above, converts the position d′ into the coordinates in aprojected image using the corrected projective transformation matrix H₁′or H₂′, places the drawing information at the converted coordinates inthe projected image, and causes the projection unit 20 to project thedrawing information. As a result, suppressing the occurrence of amismatch between the position d′ of the digital pen 50 on the drawingsurface S′ and the position of the drawing information projected on thedrawing surface S′ can be achieved as in the embodiment above.

<4.4. Fourth Modification>

The above embodiment assumes that the camera position C and theprojector position P coincide and the optical axes thereof alsocoincide. In contrast, the present modification is an example in whichthe occurrence of a mismatch between the position d′ of the digital pen50 on the drawing surface S′ and the position of drawing informationprojected on the drawing surface S′ is suppressed even when the cameraposition C and the projector position P differ from each other and/orthe optical axes of the imaging unit 10 and the projection unit 20differ from each other. Note that the present modification assumes thatthe reference point is the camera position C, as an example.

(1) Overview

FIG. 15 is a diagram for providing an overview of a drawing system 1according to the fourth modification. As shown in FIG. 15 , in thepresent modification, the camera position C and the projector position Pdiffer from each other, and the optical axes of the imaging unit 10 andthe projection unit 20 differ from each other. In the presentmodification, the calibration is performed on two calibration planes: afirst calibration plane S₁ and a second calibration plane S₂(corresponding to another calibration plane). In the example illustratedin FIG. 15 , the first calibration plane S₁ is the tabletop surface 60,the second calibration plane S₂ is the floor surface 62. Furthermore,the drawing surface S′ is a surface of any other real object and, in theexample in FIG. 15 , the drawing surface S′ is a bench seat surface 66.Note that it is assumed that the first calibration plane S₁, the secondcalibration plane S₂, and the drawing surface S′ are all parallel to oneanother. The distance between the camera position C and the firstcalibration plane S₁ is represented by h₁, the distance between theprojector position P and the first calibration plane S₁ is representedby h₂, and the distance between the first calibration plane S₁ and thesecond calibration plane S₂ is represented by h₃.

In the example illustrated in FIG. 15 , it is assumed that the digitalpen 50 is located at a position d′ on the drawing surface S′. Inaddition, the intersection point between the straight line that connectsthe camera position C with the position d′ of the digital pen 50 and thefirst calibration plane S₁ is denoted as a position d₁. In addition, theintersection point between the straight line that connects the cameraposition C with the position d′ of the digital pen 50 and the secondcalibration plane S₂ is denoted as a position d₂. In this case, thecoordinates of the digital pen 50 in a captured image are the sameregardless of whether the digital pen 50 is located at the position d₁on the first calibration plane S₁, or at the position d₂ on thecalibration plane S₂, or at the position d′ on the drawing surface S′.

The intersection point between an imaging optical axis 13, which is theoptical axis of the imaging unit 10, and the first calibration plane S₁is represented by C_(s1), and the intersection point between the imagingoptical axis 13 and the second calibration plane S₂ is represented byC_(s2). Note that, for convenience of notation, S₁ to be indicated by asubscript is expressed as s₁. The same applies to S₂. These intersectionpoints are center points in the imaging angular field of view 11. Theintersection point between a projection optical axis 23, which is theoptical axis of the projection unit 20, and the first calibration planeS₁ is represented by P_(s1), and the intersection point between theprojection optical axis 23 and the second calibration plane S₂ isrepresented by P_(s2). These intersection points are center points inthe projection angular field of view 21. The size relationship betweenthese angular fields of view and the positional relationship between thecenters of these angular fields of view are different between the firstcalibration plane S₁ and the second calibration plane S₂. This matter isexplained below with reference to FIGS. 16 to 18 .

FIG. 16 is a diagram illustrating the angular fields of view and apositional relationship between center points of the angular fields ofview on the first calibration plane S₁ illustrated in the example inFIG. 15 . FIG. 17 is a diagram illustrating the angular fields of viewand a positional relationship between center points of the angularfields of view on the second calibration plane S₂ illustrated in theexample in FIG. 15 . FIG. 18 is a diagram illustrating a relativepositional relationship between center points of angular fields of viewon each of the first calibration plane S₁ and the second calibrationplane S₂ illustrated in the example in FIG. 15 . Comparing FIG. 16 withFIG. 17 , the position d₁ of the digital pen 50 on the first calibrationplane S₁ matches the position d₂ of the digital pen 50 on the secondcalibration plane S₂. In addition, comparing FIG. 16 with FIG. 17 , thesize of the projection angular field of view 21 relative to the imagingangular field of view 11 is different between the first calibrationplane S₁ and the second calibration plane S₂. On the other hand,comparing FIG. 16 with FIG. 17 , the positional relationship between thecenter of the imaging angular field of view C_(S1) and the center of theprojection angular field of view P_(s1) on the first calibration planeS₁ differs from the positional relationship between the center of theimaging angular field of view C_(S2) and the center of the projectionangular field of view P_(s2) on the second calibration plane S₂.Therefore, as illustrated in FIG. 18 , a two-dimensional vectorv_P_(S1)C_(S1) indicating a physical mismatch between the center pointsof the angular fields of view on the first calibration plane S₁ differsfrom a two-dimensional vector v_P_(S2)C_(S2) indicating a physicalmismatch between the center points of the angular fields of view on thesecond calibration plane S₂. Note that a combination of a referencecharacter and a subscript is herein represented by a referencecharacter, an underscore, and a subscript.

The role of the projective transformation matrix H₁ obtained throughcalibration on the first calibration plane S₁ is to achieve a coordinatetransformation ensuring that the imaging angular field of view 11 fitsinto the projection angular field of view 21 on the first calibrationplane S₁. However, as illustrated in FIGS. 16 to 18 , both the sizerelationship between the imaging angular field of view 11 and theprojection angular field of view 21 and the positional relationshipbetween the center point of the imaging angular field of view 11 and thecenter point of the projection angular field of view 21 differ betweenthe first calibration plane S₁ and the second calibration plane S₂. Thatis, as a result of a change in distance from the camera position C, thescaling relationship between the angular fields of view varies and, atthe same time, the positional relationship between the centers of theimaging angular field of view and the projection angular field of viewvaries in translational movement. Therefore, on a plane other than thefirst calibration plane S₁, it is difficult to perform an appropriatecoordinate transformation only with the projective transformation matrixH₁ and, furthermore, an algorithm reflecting the scaling component onlyas in the above-described embodiment leaves room for accuracyimprovement.

Therefore, in the present modification, a translational movementcomponent on the drawing surface S′ with respect to the projectivetransformation matrix H₁ is estimated on the basis of the projectivetransformation matrix H₁ on the first calibration plane S₁ and theprojective transformation matrix H₂ on the second calibration plane S₂.Furthermore, in the present modification, a scaling component on thedrawing surface S′ with respect to the projective transformation matrixH₁ is estimated by using a method similar to the method according to theembodiment described above. The translational movement component and thescaling component on the drawing surface S′ with respect to theprojective transformation matrix H₁ obtained as described above allowfor an appropriate coordinate transformation on any drawing surface S′.That is, suppressing the occurrence of a mismatch between the positiond′ of the digital pen 50 on the drawing surface S′ and the position ofthe drawing information projected on the drawing surface S′ can beachieved.

Note that the projective transformation matrix includes a scalingcomponent and a translational movement component (=v_P_(S1)C_(S1)). Theabove embodiment assumes that the camera position C and the projectorposition P coincide and the optical axes thereof also coincide, and thisassumption means that the translational movement component(=v_P_(S1)C_(S1)) is zero.

(2) Example Configuration

An example configuration of the drawing system 1 according to thepresent modification is similar to the example configuration describedwith reference to FIG. 4 .

(3) Technical Features

Calibration

The calibration unit 42 generates a projective transformation matrix H₁related to the first calibration plane S₁. In addition, the calibrationunit 42 generates a projective transformation matrix H₂ related to thesecond calibration plane S₂.

Generating Map M₁

The map generation unit 43 generates a map M₁ that defines, with regardto each coordinate point in a captured image, the distance from thecamera position C to the first calibration plane S₁. The map generationunit 43 may generate a map M₂ that defines, with regard to eachcoordinate point in a captured image, the distance from the cameraposition C to the second calibration plane S₂.

Projection Control

The output control unit 44 according to the present modificationcontrols projection of the drawing information, further on the basis ofthe coordinates of the intersection point between the imaging opticalaxis 13 and each of the first calibration plane S₁ and the secondcalibration plane S₂ and the coordinates of the intersection pointbetween the projection optical axis 23 and each of the first calibrationplane S₁ and the second calibration plane S₂. In other words, the outputcontrol unit 44 controls projection of the drawing information on thebasis of the coordinates of the center point of the imaging angularfield of view 11 and the coordinates of the center point of theprojection angular field of view 21 on each of the first calibrationplane S₁ and the second calibration plane S₂ (that is, C_(s1), C_(s2),P_(s1), and P_(s2)).

Specifically, the output control unit 44 calculates a differentialv_(offset) of the two-dimensional vector v_P_(S2)C_(S2) relative to thetwo-dimensional vector v_P_(S1)C_(S1) in accordance with the followingequation.v _(offset) =H ₂ −H ₁  (9)

Moreover, the output control unit 44 compares the distance from thecamera position C to the first calibration plane S₁ with the distancefrom the camera position C to the second calibration plane S₂.Specifically, the output control unit 44 compares the distance from thecamera position C to a first point on the first calibration plane S₁with the distance from the camera position C to a second point on thesecond calibration plane S₂, where the first point and the second pointhave the same coordinates in a captured image. For example, assumingthat the comparison is made through a ratio calculation, the ratioinformation (that is, the ratio) k is defined by the following equation.k=(h ₁ +h ₃)/h ₁  (10)

Here, the distances from the camera position C to the first point andthe second point, respectively, are obtained by, for example, using theradii of the light-emitting circles provided when the digital pen 50 islocated at the first point and the second point, respectively. Lettingr(d₁) be the radius provided when the digital pen 50 is located at thefirst point d₁ and letting r(d₂) be the radius provided when the digitalpen 50 is located at the second point d₂, the ratio information k isobtained by the following equation.k=r(d ₁)/r(d ₂)  (11)

Each of r(d₁) and r(d₂) may be acquired by causing the user to locatethe digital pen 50 at each of the first point and the second point. Forexample, first, when the user locates the digital pen 50 at any point onthe first calibration plane S₁, the output control unit 44 stores thepoint as the first point and acquires the radius r(d₁) of thelight-emitting circle at the first point. Next, the output control unit44 controls projection of information onto the second calibration planeS₂, the information being intended for identifying the second point. Forexample, the output control unit 44 sequentially causes points of lightobservable by the imaging unit 10 (visible light, for example) to beprojected on the second calibration plane S₂ at various coordinatepoints. Then, the output control unit 44 identifies the second point bydetecting the point of light projected at the same coordinates as thefirst point in a captured image. Such an identification method is alsocalled a sequential search. Then, the output control unit 44 acquiresthe radius r(d₂) of the light-emitting circle at the second point bycausing guide information indicating the identified second point to beprojected on the plane so that the user locates the digital pen 50 atthe second point.

On the other hand, r(d₁) may be acquired by referring to the coordinatesof the first point in a captured image on the map M₁. In this case, theuser operation of locating the digital pen 50 at the first point isomitted. Likewise, if the map M₂ is generated, r(d₂) may be acquired byreferring to the coordinates of the second point in a captured image onthe map M₂. In this case, the user operation of locating the digital pen50 at the second point is omitted.

On the basis of the ratio information k obtained by the above-describedcomparison, the output control unit 44 controls projection of thedrawing information. Specifically, the ratio information k reveals therelationship in which a translational movement by v_(offset) occurs whenthe distance along a depth direction from the camera position C to thedigital pen 50 becomes k times the distance from the camera position Cto the first calibration plane S₁. Therefore, the output control unit 44uses this relationship to calculate a projective transformation matrixH′ on the drawing surface S′, and uses the projective transformationmatrix H′ to control the coordinates of the drawing information. Thefollowing describes a method for calculating the projectivetransformation matrix H′ in detail.

Assuming that h₁=1, the above mathematical expression (10) istransformed into the following equation.k=1+h ₃  (12)

The assumption that h₁=1 is equivalent to setting the radius r(d₁) ofthe light-emitting circle on the map M₁ to 1. As a result, the map M₂ ofthe second calibration plane S₂ is a map with the light-emitting circlewhose radius is at a certain ratio relative to the radius r(d₁) of thelight-emitting circle on the map M₁. That is, the map M₂ is obtained inaccordance with the following equation.M ₂=(1/k)×M ₁  (13)

When the user locates the digital pen 50 at any position d′ on thedrawing surface S′, the position acquisition unit 41 acquires the radiusr(d′) of the light-emitting circle at the position d′. Then, the outputcontrol unit 44 refers to the map M₁ to acquire the radius r(d₁) of thelight-emitting circle at the coordinates (x_(d), y_(d))=(x_(d′),y_(d′)). Then, on the basis of the acquired r(d₁), the output controlunit 44 calculates the ratio k′ between the first distance and thesecond distance in accordance with the following equation, which issimilar to the mathematical expression (5) above.k′=r(d ₁)/r(d′)  (14)

Next, the output control unit 44 calculates, in accordance with thefollowing equation, a differential (v_(offset))′ of the two-dimensionalvector v_P_(S2′), C_(S2′) relative to the two-dimensional vectorv_P_(S1)C_(S1), the differential representing a physical mismatchbetween the centers of the angular fields of view on the drawing surfaceS′.(v _(offset))′=(k′/k)×v _(offset)  (15)

Then, the output control unit 44 calculates a projective transformationmatrix H′ on the drawing surface S′ in accordance with the followingequation.H′=H ₁+(v _(offset))′  (16)

Supplementary Information

Note that the algorithm of the present modification is based on h₁=h₂.However, when h₁≠h₂, the algorithm of the present modification can bestill applied by estimating h₂ through distance measurement orcalibration.

In addition, the drawing surface S′ may be a curved surface. This matteris explained below with reference to FIG. 19 .

FIG. 19 is a diagram for explaining an example use of the drawing system1 according to the present modification. As illustrated in FIG. 19 , adoll 64 is placed on the tabletop surface 60, and a surface of the doll64 serves as the drawing surface S′. As illustrated in FIG. 19 , thesurface of the doll 64 is curved. The drawing surface S′ may be asurface of a three-dimensional object in any shape having a curvedsurface and the like. However, the above-described algorithm assumesthat the first calibration plane S₁, the second calibration plane S₂,and the drawing surface S′ are parallel. For this reason, in a casewhere the drawing surface S′ is curved, the area of the position d′ ofthe digital pen 50 in the drawing surface S′ is segmentalized so thatthe area is regarded as parallel to the first calibration plane S₁ andthe second calibration plane S₂, and then the above-described algorithmis applied.

5. EXAMPLE HARDWARE CONFIGURATION

Finally, a hardware configuration of the information processing deviceaccording to the present embodiment will be described with reference toFIG. 20 . FIG. 20 is a block diagram illustrating an example hardwareconfiguration of the information processing device according to thepresent embodiment. Note that an information processing device 900illustrated in FIG. 20 can implement the drawing system 1 illustrated inFIGS. 4, 12, and 13 , for example. Information processing by the drawingsystem 1 according to the present embodiment is implemented by softwareand the hardware described below in collaboration with each other.

As illustrated in FIG. 20 , the information processing device 900includes a central processing unit (CPU) 901, a read only memory (ROM)902, a random access memory (RAM) 903, and a host bus 904 a.Furthermore, the information processing device 900 includes a bridge904, an external bus 904 b, an interface 905, an input device 906, anoutput device 907, a storage device 908, a drive 909, a connection port911, and a communication device 913. The information processing device900 may include a processing circuit, such as an electric circuit, aDSP, or an ASIC, in place of or in addition to the CPU 901.

The CPU 901, which functions as an arithmetic processing device and acontrol device, controls the overall operation in the informationprocessing device 900 in accordance with various programs. Furthermore,the CPU 901 may be a microprocessor. The ROM 902 stores programs,operation parameters, and the like to be used by the CPU 901. The RAM903 temporarily stores programs to be used during the execution by theCPU 901, parameters that appropriately vary during the execution, andthe like. The CPU 901 may be included in, for example, the projectioncontrol unit 40 illustrated in FIGS. 4, 12, and 13 .

The CPU 901, the ROM 902, and the RAM 903 are connected to one anotherby the host bus 904 a including a CPU bus or the like. The host bus 904a is connected to the external bus 904 b such as a peripheral componentinterconnect/interface (PCI) bus via the bridge 904. Note that the hostbus 904 a, the bridge 904, and the external bus 904 b may notnecessarily be included separately but these functions may beimplemented in a single bus.

The input device 906 may include a device that detects informationregarding a drawing tool. For example, the input device 906 may includevarious sensors such as an image sensor (for example, a camera), a depthsensor (for example, a stereo camera), an acceleration sensor, a gyrosensor, a geomagnetic sensor, an optical sensor, a sound sensor, adistance measuring sensor, and a force sensor. Furthermore, the inputdevice 906 may acquire information regarding the state of the drawingtool itself such as the attitude and moving speed of the drawing tool,and information regarding the surrounding environment of the informationprocessing device 900 such as illuminance and noise around the drawingtool. In addition, the input device 906 may include a Global NavigationSatellite System (GNSS) module that receives a GNSS signal (for example,a global positioning system (GPS) signal from a GPS satellite) from aGNSS satellite and measures the position information including thelatitude, longitude, and altitude of the device. Furthermore, regardingthe position information, the input device 906 may detect positionsthrough transmission and reception to and from Wi-Fi (registeredtrademark), a mobile phone, a PHS, a smart phone, or the like, orthrough short-range communication or the like. The input device 906 mayinclude, for example, the imaging unit 10 shown in FIG. 4 , the distancemeasuring unit 12 shown in FIG. 12 , and the inertial sensor unit 16shown in FIG. 13 .

The output device 907 includes a device that can visually or audiblygive notification of the acquired information to the user. Examples ofsuch a device include display devices such as a laser projector, an LEDprojector, and a lamp, sound output devices such as a speaker and aheadphone, printer devices, and the like. The output device 907 outputs,for example, results obtained by the information processing device 900performing various types of processing. Specifically, the display devicevisually displays results obtained by the information processing device900 performing various types of processing in various forms such astext, images, tables, graphs, and the like. On the other hand, the audiooutput device converts an audio signal including the reproduced audiodata, acoustic data, and the like into an analog signal, and audiblyoutputs the analog signal. The aforementioned display device may beincluded in, for example, the projection unit 20 shown in FIGS. 4, 12,and 13 .

The storage device 908 is a data storage device formed as an example ofthe storage unit in the information processing device 900. The storagedevice 908 is implemented by, for example, a magnetic storage devicesuch as an HDD, a semiconductor storage device, an optical storagedevice, or a magneto-optical storage device. The storage device 908 mayinclude a storage medium, a recording device that records data on thestorage medium, a reading device that reads data from the storagemedium, a deletion device that deletes data recorded on the storagemedium, and the like. The storage device 908 stores programs to beexecuted by the CPU 901 and various types of data, as well as varioustypes of data acquired from the outside and other data. The storagedevice 908 may be included in, for example, the storage unit 30 shown inFIGS. 4, 12, and 13 .

The drive 909 is a reader/writer for a storage medium, and is built inor externally attached to the information processing device 900. Thedrive 909 reads information recorded on the attached removable storagemedium, such as a magnetic disk, an optical disk, a magneto-opticaldisk, or a semiconductor memory, and outputs the information to the RAM903. Furthermore, the drive 909 is capable of writing information to theremovable storage medium.

The connection port 911, which is an interface connected to an externaldevice, is a connection port connected to an external device and iscapable of transmitting data by Universal Serial Bus (USB), for example.

The communication device 913 is, for example, a communication interfaceincluding a communication device or the like for connecting to thenetwork 920. The communication device 913 is, for example, acommunication card or the like for a wired or wireless local areanetwork (LAN), Long Term Evolution (LTE), Bluetooth (registeredtrademark), or wireless USB (WUSB). Alternatively, the communicationdevice 913 may be a router for optical communication, a router forasymmetric digital subscriber line (ADSL), a modem for various types ofcommunication, or the like. The communication device 913 is capable oftransmitting and receiving signals and the like to and from, forexample, the Internet or another communication device in accordance witha predetermined protocol such as TCP/IP. The communication device 913can be used for communication with the projection control unit 40 andthe digital pen 50.

Note that the network 920 is a wired or wireless transmission path forinformation transmitted from a device connected to the network 920. Forexample, the network 920 may include a public line network such as theInternet, a telephone line network, or a satellite communicationnetwork, any of various local area networks (LANs) including Ethernet(registered trademark) or wide area networks (WANs), and the like.Furthermore, the network 920 may include a private line network such asan Internet Protocol-Virtual Private Network (IP-VPN).

The foregoing has described an example of a hardware configuration thatcan implement functions of the information processing device 900according to the present embodiment. Each of the above-describedcomponents may be implemented by using a general-purpose member, or maybe implemented by using the hardware specialized for the functions ofeach of the components. Therefore, the hardware configuration to be usedcan be changed as appropriate in accordance with the technical level onan occasion of carrying out the present embodiment.

Note that it is possible to create a computer program for achieving thefunctions of the information processing device 900 according to thepresent embodiment as described above, and implement the computerprogram on a PC or the like. Furthermore, it is also possible to providea computer-readable recording medium containing such a computer program.The recording medium is, for example, a magnetic disk, an optical disk,a magneto-optical disk, a flash memory, or the like. Furthermore, theabove-described computer program may be distributed via, for example, anetwork without using a recording medium.

6. CONCLUSION

An embodiment of the present disclosure has been described above indetail with reference to FIGS. 1 to 20 . As described above, theprojection control unit 40 according to the present embodiment controlsprojection of drawing information performed by the projection unit 20 onthe basis of the ratio information between a first distance and a seconddistance, the first distance being from the reference point to thedigital pen 50 as acquired from sensing information regarding thedigital pen 50, and the second distance being from the reference pointto the intersection point between the straight line connecting thereference point with the digital pen 50 and the calibration plane.According to the present embodiment, the ratio information between thefirst distance and the second distance is used for projection control ofthe drawing information. More specifically, on the basis of the ratioinformation between the first distance and the second distance, aprojective transformation matrix H′ specialized for the position d′ ofthe digital pen 50 in drawing operation on the drawing surface S′ isgenerated in real time from a projective transformation matrix H on thecalibration plane S. As a result, it is made possible to suppress theoccurrence of a mismatch between the position of the digital pen 50 onthe drawing surface S′ and the position of the drawing informationprojected on the drawing surface S′, the mismatch occurring when theprojective transformation matrix H on the calibration plane S is used asit is. Since the projective transformation matrix H′ is generated inreal time, the present technology is effective when, for example, animage is projected onto a surface of a moving real object. Furthermore,since it is unnecessary to perform a calibration for each drawingsurface S′, the present technology is also effective for projection on anon-flat, curved or uneven surface. Furthermore, since the projectorposition P is not involved in the calculation of the projectivetransformation matrix H′, the present technology can be applied to thecase where the projector position P is unknown.

The occurrence of a mismatch of positions described above causes asignificant trouble in a drawing operation or the like of characters orpictures, the operation being continued with reference to the trailproduced immediately before the drawing. In this regard, according tothe present embodiment, it is made possible to provide a comfortabledrawing experience to the user because the occurrence of a mismatch ofpositions is suppressed.

Furthermore, in the present embodiment, at least one of the size, thebrightness, or the color of drawing information may be controlled, inaddition to the coordinates of the drawing information. Therefore, theresult of drawing on various places in the real space with the digitalpen 50 can be made look like the drawing made with an analog pen. Forexample, to consider a color of a carpet and tablecloth to be placed ina room, when the user paints the same color thereon with the digital pen50, the user will have difficulty in deciding a color if the carpet andtablecloth appear in a different color. In this regard, the presentembodiment allows the user to make a practical study more smoothlybecause a certain color painted on any place in the real space appearsin that color.

Preferred embodiments of the present disclosure have been describedabove in detail with reference to the accompanying drawings, but thetechnical scope of the present disclosure is not limited to theseexamples. It is apparent that a person having ordinary knowledge in thetechnical field of the present disclosure can arrive at various changesor modifications within the scope of the technical idea described in theclaims, and it is naturally understood that these changes andmodifications belong to the technical scope of the present disclosure.

For example, the mapping of the drawing system 1 described in the aboveembodiment to devices may be done in various possible ways. For example,the storage unit 30 and the projection control unit 40 may be formed asa single information processing device, or may be disposed in a serveron a cloud. Alternatively, the storage unit 30 and the projectioncontrol unit 40 may be disposed on the imaging unit 10, on theprojection unit 20, or on the digital pen 50.

Furthermore, process steps described herein with reference to flowchartsand sequence diagrams may not necessarily be carried out in the order asillustrated. Some processing steps may be performed in parallel.Furthermore, additional process steps may be employed, and some processsteps may be omitted.

Furthermore, the effects described herein are merely illustrative orexemplary effects, and are not restrictive. That is, in addition to orin place of the effects described above, the technology according to thepresent disclosure can provide other effects that are obvious to thoseskilled in the art from the descriptions herein.

Note that the following configurations also belong to the technicalscope of the present disclosure.

(1)

An information processing device including:

a projection control unit that controls projection of drawinginformation, the projection being performed by a projection device, onthe basis of ratio information between a first distance and a seconddistance, the first distance being acquired from sensing informationregarding a drawing tool and being from a reference point to the drawingtool, and the second distance being from the reference point to anintersection point between a straight line connecting the referencepoint with the drawing tool and a calibration plane.

(2)

The information processing device according to (1), in which the ratioinformation is based on a ratio between the first distance and thesecond distance.

(3)

The information processing device according to (1) or (2), in which theprojection control unit controls coordinates of the drawing informationin a projected image projected by the projection device.

(4)

The information processing device according to any one of (1) to (3), inwhich the projection control unit controls a size of the drawinginformation in a projected image projected by the projection device.

(5)

The information processing device according to any one of (1) to (4), inwhich the projection control unit controls brightness of the drawinginformation in a projected image projected by the projection device.

(6)

The information processing device according to any one of (1) to (5), inwhich the projection control unit controls a color of the drawinginformation in a projected image projected by the projection device.

(7)

The information processing device according to any one of (1) to (6), inwhich the calibration plane is a plane of a surface of a calibrationtarget object and a plane obtained by virtually extending the plane ofthe surface of the calibration target object.

(8)

The information processing device according to any one of (1) to (7), inwhich the sensing information includes a captured image, and a positionof the reference point includes a position of an imaging device thatcaptures the captured image.

(9)

The information processing device according to (8), in which theprojection control unit acquires the first distance on the basis of asize of a specific portion of the drawing tool, the specific portionappearing in the captured image.

(10)

The information processing device according to (9), in which theprojection control unit acquires the first distance on the basis of asize of a shape of light emitted from the drawing tool, the shapeappearing in the captured image.

(11)

The information processing device according to any one of (8) to (10),in which the projection control unit generates a map that defines adistance from the reference point to the calibration plane with respectto each coordinate point in the captured image, and acquires, as thesecond distance, a distance defined in the map with respect to acoordinate point corresponding to the coordinate point of the drawingtool included in the captured image.

(12)

The information processing device according to (11), in which the map isgenerated on the basis of a size of a specific portion of the drawingtool, the specific portion appearing in the captured image, in a casewhere the drawing tool is located at a plurality of points on thecalibration plane.

(13)

The information processing device according to (11), in which the map isgenerated on the basis of a distance to the drawing tool, the distancebeing acquired by a distance measuring sensor, in a case where thedrawing tool is located at a plurality of points on the calibrationplane.

(14)

The information processing device according to any one of (11) to (13),in which the projection control unit controls projection of the drawinginformation further on the basis of coordinates of an intersection pointbetween an optical axis of the imaging device and each of thecalibration plane and another calibration plane, and coordinates of anintersection point between an optical axis of the projection device andeach of the calibration plane and the another calibration plane.

(15)

The information processing device according to (14), in which theprojection control unit controls projection of the drawing informationon the basis of the ration information between a distance from thereference point to a first point on the calibration plane and a distancefrom the reference point to a second point on the another calibrationplane, the first point and the second point having the same coordinateson the captured image.

(16)

The information processing device according to (15), in which theprojection control unit controls projection of information foridentifying the second point onto the another calibration plane.

(17)

The information processing device according to any one of (1) to (7), inwhich the projection control unit controls projection of the drawinginformation, the projection being performed by the projection device, onthe basis of a three-dimensional position of the drawing tool in a casewhere the drawing tool is located at a plurality of known points on aprojected image projected on the calibration plane and the anothercalibration plane and a three-dimensional position of the drawing toolin drawing operation.

(18)

The information processing device according to (17), in which theprojection control unit generates a projective transformation matrix forconverting the three-dimensional position on the calibration plane intocoordinates in a projected image projected by the projection device, onthe basis of a three-dimensional position of the drawing tool in a casewhere the drawing tool is located at a plurality of known points in aprojected image projected on the calibration plane and the anothercalibration plane.

(19)

An information processing method including:

controlling projection of drawing information, the projection beingperformed by a projection device, on the basis of ratio informationbetween a first distance and a second distance, the first distance beingacquired from sensing information regarding a drawing tool and beingfrom a reference point to the drawing tool, and the second distancebeing from the reference point to an intersection point between astraight line connecting the reference point with the drawing tool and acalibration plane.

(20)

A recording medium recording a program that causes a computer tofunction as:

a projection control unit that controls projection of drawinginformation, the projection being performed by a projection device, onthe basis of ratio information between a first distance and a seconddistance, the first distance being acquired from sensing informationregarding a drawing tool and being from a reference point to the drawingtool, and the second distance being from the reference point to anintersection point between a straight line connecting the referencepoint with the drawing tool and a calibration plane.

REFERENCE SIGNS LIST

-   1 Drawing system-   10 Imaging unit-   11 Imaging angular field of view-   12 Distance measuring unit-   16 Inertial sensor unit-   20 Projection unit-   21 Projection angular field of view-   30 Storage unit-   40 Projection control unit-   41 Position acquisition unit-   42 Calibration unit-   43 Map generation unit-   44 Output control unit-   50 Digital pen-   60 Tabletop surface-   61 Auxiliary board-   62 Floor surface-   64 Doll-   66 Bench seat surface

The invention claimed is:
 1. An information processing device,comprising: a projection control unit configured to control a projectiondevice to project drawing information based on a three-dimensionalposition of a drawing tool in a drawing operation and ratio informationbetween a first distance and a second distance, wherein the drawing toolis at a plurality of points of an image projected on a first calibrationplane and a second calibration plane located parallel to the firstcalibration plane, the first distance is acquired from sensinginformation regarding the drawing tool, the first distance is a distancefrom a reference point to the drawing tool, and the second distance is adistance from the reference point to an intersection point between astraight line connecting the reference point with the drawing tool andthe first calibration plane.
 2. The information processing deviceaccording to claim 1, wherein the ratio information is based on a ratiobetween the first distance and the second distance.
 3. The informationprocessing device according to claim 1, wherein the projection controlunit is further configured to control coordinates of the drawinginformation in the image projected by the projection device.
 4. Theinformation processing device according to claim 1, wherein theprojection control unit is further configured to control a size of thedrawing information in a projected image projected by the projectiondevice.
 5. The information processing device according to claim 1,wherein the projection control unit is further configured to controlbrightness of the drawing information in the image projected by theprojection device.
 6. The information processing device according toclaim 1, wherein the projection control unit is further configured tocontrol a color of the drawing information in the image projected by theprojection device.
 7. The information processing device according toclaim 1, wherein the first calibration plane is a plane of a surface ofa calibration target object and a plane obtained by virtually extendingthe plane of the surface of the calibration target object.
 8. Theinformation processing device according to claim 1, wherein the sensinginformation includes a captured image, and a position of the referencepoint includes a position of an imaging device that captures thecaptured image.
 9. The information processing device according to claim8, wherein the projection control unit is further configured to acquirethe first distance based on a size of a specific portion of the drawingtool, and the specific portion appears in the captured image.
 10. Theinformation processing device according to claim 9, wherein theprojection control unit is further configured to acquire the firstdistance based on a size of a shape of light emitted from the drawingtool, and the shape appears in the captured image.
 11. The informationprocessing device according to claim 8, wherein the projection controlunit is further configured to: generate a map that defines a distancefrom the reference point to the calibration plane with respect to eachcoordinate point in the captured image; and acquire, as the seconddistance, a distance defined in the map with respect to a coordinatepoint corresponding to a coordinate point of the drawing tool includedin the captured image.
 12. The information processing device accordingto claim 11, wherein the map is generated based on a size of a specificportion of the drawing tool in a case where the drawing tool is at aplurality of points on the first calibration plane, and the specificportion appears in the captured image.
 13. The information processingdevice according to claim 11, wherein the map is generated based on adistance to the drawing tool in a case where the drawing tool at aplurality of points on the first calibration plane, and a distancemeasuring sensor acquires the distance.
 14. The information processingdevice according to claim 11, wherein the projection control unit isfurther configured to control projection of the drawing informationbased on coordinates of an intersection point between an optical axis ofthe imaging device and each of the first calibration plane and thesecond calibration plane, and coordinates of an intersection pointbetween an optical axis of the projection device and each of the firstcalibration plane and the second calibration plane.
 15. An informationprocessing method, comprising: controlling a projection device toproject drawing information based on a three-dimensional position of adrawing tool in a drawing operation and ratio information between afirst distance and a second distance, wherein the drawing tool is at aplurality of points of an image projected on a first calibration planeand a second calibration plane located parallel to the first calibrationplane, the first distance is acquired from sensing information regardingthe drawing tool, the first distance is a distance from a referencepoint to the drawing tool, and the second distance is a distance fromthe reference point to an intersection point between a straight lineconnecting the reference point with the drawing tool and the firstcalibration plane.
 16. The information processing device according toclaim 15, wherein the projection control unit is further configured tocontrol projection of information for identifying the second point ontothe second calibration plane.
 17. The information processing deviceaccording to claim 1, wherein the projection control unit is furtherconfigured to generate a projective transformation matrix for convertingthe three-dimensional position on the calibration plane into coordinatesin the image projected by the projection device, and the projectivetransformation matrix is projected based on the three-dimensionalposition of the drawing tool in a case where the drawing tool is at theplurality of points in the image projected on the first calibrationplane and the second calibration plane.
 18. An information processingmethod, comprising: controlling a projection device to project ofdrawing information based on a three-dimensional position of a drawingtool in a drawing operation and ratio information between a firstdistance and a second distance, wherein the drawing tool is at aplurality of points on an image projected on a first calibration planeand a second calibration plane, the first distance is acquired fromsensing information regarding the drawing tool, the first distance is adistance from a reference point to the drawing tool, and the seconddistance is a distance from the reference point to an intersection pointbetween a straight line connecting the reference point with the drawingtool and the first calibration plane.
 19. A non-transitorycomputer-readable medium having stored thereon, computer-executableinstructions which, when executed by a computer, cause the computer toexecute operations, the operations comprising: controlling a projectiondevice to project drawing information based on a three-dimensionalposition of a drawing tool in a drawing operation and ratio informationbetween a first distance and a second distance, wherein the drawing toolis at a plurality of points of an image projected on a first calibrationplane and a second calibration plane located parallel to the firstcalibration plane, the first distance is acquired from sensinginformation regarding the drawing tool, the first distance is a distancefrom a reference point to the drawing tool, and the second distance is adistance from the reference point to an intersection point between astraight line connecting the reference point with the drawing tool andthe first calibration plane.